How Causal Heterogeneity Can Influence Statistical Significance in Clinical Trials

Finding that an association is statistically insignificant in a clinical trial has two distinct explanations. (1) The association may be real, but the sample size is too small to distinguish the results from those due to chance. (2) The association is spurious – a coincidence due to chance. Based on Weisberg (2011), this paper argues that there is a third explanation: the association may be real but the mixture of potential outcomes – causal heterogeneity – may give results that are indistinguishable from those due to chance. These potential outcomes involve counterfactuals: outcomes that could have happened but did not. Causal heterogeneity exists when a treatment interacts to give different results with different units of a population. This paper uses a model of causal heterogeneity that is generally accessible. Statistical education should extend introductory statistics courses so they (a) show how potential outcomes affect statistical significance in clinical trials and (b) highlight the importance of causally-heterogeneous subgroups in determining whether a treatment effect is statistically significant.